Artificial Intelligence Nanodegree

Machine Translation Project

In this notebook, sections that end with '(IMPLEMENTATION)' in the header indicate that the following blocks of code will require additional functionality which you must provide. Please be sure to read the instructions carefully!

Introduction

In this notebook, you will build a deep neural network that functions as part of an end-to-end machine translation pipeline. Your completed pipeline will accept English text as input and return the French translation.

  • Preprocess - You'll convert text to sequence of integers.
  • Models Create models which accepts a sequence of integers as input and returns a probability distribution over possible translations. After learning about the basic types of neural networks that are often used for machine translation, you will engage in your own investigations, to design your own model!
  • Prediction Run the model on English text.

Dataset

We begin by investigating the dataset that will be used to train and evaluate your pipeline. The most common datasets used for machine translation are from WMT. However, that will take a long time to train a neural network on. We'll be using a dataset we created for this project that contains a small vocabulary. You'll be able to train your model in a reasonable time with this dataset.

Load Data

The data is located in data/small_vocab_en and data/small_vocab_fr. The small_vocab_en file contains English sentences with their French translations in the small_vocab_fr file. Load the English and French data from these files from running the cell below.

In [1]:
import helper


# Load English data
english_sentences = helper.load_data('data/small_vocab_en')
# Load French data
french_sentences = helper.load_data('data/small_vocab_fr')

print('Dataset Loaded')
Dataset Loaded

Files

Each line in small_vocab_en contains an English sentence with the respective translation in each line of small_vocab_fr. View the first two lines from each file.

In [2]:
print(len(french_sentences))
print(french_sentences[:1])
print(english_sentences[:1])
for sample_i in range(2):
    print('small_vocab_en Line {}:  {}'.format(sample_i + 1, english_sentences[sample_i]))
    print('small_vocab_fr Line {}:  {}'.format(sample_i + 1, french_sentences[sample_i]))
137861
["new jersey est parfois calme pendant l' automne , et il est neigeux en avril ."]
['new jersey is sometimes quiet during autumn , and it is snowy in april .']
small_vocab_en Line 1:  new jersey is sometimes quiet during autumn , and it is snowy in april .
small_vocab_fr Line 1:  new jersey est parfois calme pendant l' automne , et il est neigeux en avril .
small_vocab_en Line 2:  the united states is usually chilly during july , and it is usually freezing in november .
small_vocab_fr Line 2:  les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .

From looking at the sentences, you can see they have been preprocessed already. The puncuations have been delimited using spaces. All the text have been converted to lowercase. This should save you some time, but the text requires more preprocessing.

Vocabulary

The complexity of the problem is determined by the complexity of the vocabulary. A more complex vocabulary is a more complex problem. Let's look at the complexity of the dataset we'll be working with.

In [3]:
import collections


english_words_counter = collections.Counter([word for sentence in english_sentences for word in sentence.split()])
french_words_counter = collections.Counter([word for sentence in french_sentences for word in sentence.split()])

print('{} English words.'.format(len([word for sentence in english_sentences for word in sentence.split()])))
print('{} unique English words.'.format(len(english_words_counter)))
print('10 Most common words in the English dataset:')
print('"' + '" "'.join(list(zip(*english_words_counter.most_common(10)))[0]) + '"')
print()
print('{} French words.'.format(len([word for sentence in french_sentences for word in sentence.split()])))
print('{} unique French words.'.format(len(french_words_counter)))
print('10 Most common words in the French dataset:')
print('"' + '" "'.join(list(zip(*french_words_counter.most_common(10)))[0]) + '"')
1823250 English words.
227 unique English words.
10 Most common words in the English dataset:
"is" "," "." "in" "it" "during" "the" "but" "and" "sometimes"

1961295 French words.
355 unique French words.
10 Most common words in the French dataset:
"est" "." "," "en" "il" "les" "mais" "et" "la" "parfois"

For comparison, Alice's Adventures in Wonderland contains 2,766 unique words of a total of 15,500 words.

Preprocess

For this project, you won't use text data as input to your model. Instead, you'll convert the text into sequences of integers using the following preprocess methods:

  1. Tokenize the words into ids
  2. Add padding to make all the sequences the same length.

Time to start preprocessing the data...

Tokenize (IMPLEMENTATION)

For a neural network to predict on text data, it first has to be turned into data it can understand. Text data like "dog" is a sequence of ASCII character encodings. Since a neural network is a series of multiplication and addition operations, the input data needs to be number(s).

We can turn each character into a number or each word into a number. These are called character and word ids, respectively. Character ids are used for character level models that generate text predictions for each character. A word level model uses word ids that generate text predictions for each word. Word level models tend to learn better, since they are lower in complexity, so we'll use those.

Turn each sentence into a sequence of words ids using Keras's Tokenizer function. Use this function to tokenize english_sentences and french_sentences in the cell below.

Running the cell will run tokenize on sample data and show output for debugging.

In [4]:
import project_tests as tests
from keras.preprocessing.text import Tokenizer


def tokenize(x):
    """
    Tokenize x
    :param x: List of sentences/strings to be tokenized
    :return: Tuple of (tokenized x data, tokenizer used to tokenize x)
    """
    # TODO: Implement
    tokenizer = Tokenizer()
    tokenizer.fit_on_texts(x)    
    return tokenizer.texts_to_sequences(x), tokenizer
tests.test_tokenize(tokenize)

# Tokenize Example output
text_sentences = [
    'The quick brown fox jumps over the lazy dog .',
    'By Jove , my quick study of lexicography won a prize .',
    'This is a short sentence .']
text_tokenized, text_tokenizer = tokenize(text_sentences)
print(text_tokenizer.word_index)
print()
for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):
    print('Sequence {} in x'.format(sample_i + 1))
    print('  Input:  {}'.format(sent))
    print('  Output: {}'.format(token_sent))
Using TensorFlow backend.
{'quick': 2, 'won': 16, 'by': 10, 'study': 13, 'fox': 5, 'jove': 11, 'brown': 4, 'over': 7, 'sentence': 21, 'lexicography': 15, 'this': 18, 'my': 12, 'dog': 9, 'is': 19, 'prize': 17, 'lazy': 8, 'the': 1, 'a': 3, 'short': 20, 'jumps': 6, 'of': 14}

Sequence 1 in x
  Input:  The quick brown fox jumps over the lazy dog .
  Output: [1, 2, 4, 5, 6, 7, 1, 8, 9]
Sequence 2 in x
  Input:  By Jove , my quick study of lexicography won a prize .
  Output: [10, 11, 12, 2, 13, 14, 15, 16, 3, 17]
Sequence 3 in x
  Input:  This is a short sentence .
  Output: [18, 19, 3, 20, 21]

Padding (IMPLEMENTATION)

When batching the sequence of word ids together, each sequence needs to be the same length. Since sentences are dynamic in length, we can add padding to the end of the sequences to make them the same length.

Make sure all the English sequences have the same length and all the French sequences have the same length by adding padding to the end of each sequence using Keras's pad_sequences function.

In [5]:
import numpy as np
from keras.preprocessing.sequence import pad_sequences


def pad(x, length=None):
    """
    Pad x
    :param x: List of sequences.
    :param length: Length to pad the sequence to.  If None, use length of longest sequence in x.
    :return: Padded numpy array of sequences
    """
    # TODO: Implement
    if length is None:
        length = len(max(x, key=len))
    print(max(x, key=len))
    return pad_sequences(x, maxlen=length,padding='post')

tests.test_pad(pad)  
# Pad Tokenized output
test_pad = pad(text_tokenized)
for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):
    print('Sequence {} in x'.format(sample_i + 1))
    print('  Input:  {}'.format(np.array(token_sent)))
    print('  Output: {}'.format(pad_sent))
  
[0, 1, 2, 3, 4, 5]
[0, 1, 2, 3, 4, 5]
[10, 11, 12, 2, 13, 14, 15, 16, 3, 17]
Sequence 1 in x
  Input:  [1 2 4 5 6 7 1 8 9]
  Output: [1 2 4 5 6 7 1 8 9 0]
Sequence 2 in x
  Input:  [10 11 12  2 13 14 15 16  3 17]
  Output: [10 11 12  2 13 14 15 16  3 17]
Sequence 3 in x
  Input:  [18 19  3 20 21]
  Output: [18 19  3 20 21  0  0  0  0  0]

Preprocess Pipeline

Your focus for this project is to build neural network architecture, so we won't ask you to create a preprocess pipeline. Instead, we've provided you with the implementation of the preprocess function.

In [6]:
def preprocess(x, y):
    """
    Preprocess x and y
    :param x: Feature List of sentences
    :param y: Label List of sentences
    :return: Tuple of (Preprocessed x, Preprocessed y, x tokenizer, y tokenizer)
    """
    preprocess_x, x_tk = tokenize(x)
    preprocess_y, y_tk = tokenize(y)

    preprocess_x = pad(preprocess_x)
    preprocess_y = pad(preprocess_y)

    # Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions
    preprocess_y = preprocess_y.reshape(*preprocess_y.shape, 1)

    return preprocess_x, preprocess_y, x_tk, y_tk

preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer =\
    preprocess(english_sentences, french_sentences)

print('Data Preprocessed')
[5, 20, 21, 1, 9, 62, 4, 43, 7, 3, 1, 9, 51, 2, 45]
[35, 34, 105, 103, 9, 19, 15, 111, 22, 11, 24, 6, 3, 1, 9, 226, 22, 227, 26, 11, 24]
Data Preprocessed

Models

In this section, you will experiment with various neural network architectures. You will begin by training four relatively simple architectures.

  • Model 1 is a simple RNN
  • Model 2 is a RNN with Embedding
  • Model 3 is a Bidirectional RNN
  • Model 4 is an optional Encoder-Decoder RNN

After experimenting with the four simple architectures, you will construct a deeper architecture that is designed to outperform all four models.

Ids Back to Text

The neural network will be translating the input to words ids, which isn't the final form we want. We want the French translation. The function logits_to_text will bridge the gab between the logits from the neural network to the French translation. You'll be using this function to better understand the output of the neural network.

In [7]:
def logits_to_text(logits, tokenizer):
    """
    Turn logits from a neural network into text using the tokenizer
    :param logits: Logits from a neural network
    :param tokenizer: Keras Tokenizer fit on the labels
    :return: String that represents the text of the logits
    """
    index_to_words = {id: word for word, id in tokenizer.word_index.items()}
    index_to_words[0] = '<PAD>'

    return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])

print('`logits_to_text` function loaded.')
`logits_to_text` function loaded.

Model 1: RNN (IMPLEMENTATION)

RNN A basic RNN model is a good baseline for sequence data. In this model, you'll build a RNN that translates English to French.

In [10]:
from keras.layers import GRU, Input, Dense, TimeDistributed,LSTM
from keras.models import Model, Sequential
from keras.layers import Activation
from keras.optimizers import Adam
from keras.losses import sparse_categorical_crossentropy

learning_rate=.001
def simple_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
    """
    Build and train a basic RNN on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param english_vocab_size: Number of unique English words in the dataset
    :param french_vocab_size: Number of unique French words in the dataset
    :return: Keras model built, but not trained
    """
    # TODO: Build the layers
#     print(input_shape,input_shape[1:],output_sequence_length)
    model = Sequential()
#     model.add(Embedding(english_vocab_size, french_vocab_size,input_shape=input_shape[1:]))
    model.add(LSTM(256, dropout=0.2, recurrent_dropout=0.2, return_sequences=True,input_shape=input_shape[1:]))
    model.add(TimeDistributed(Dense(french_vocab_size, activation="softmax")))
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    model.summary()
    return model
tests.test_simple_model(simple_model)


# Reshaping the input to work with a basic RNN
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))

# Train the neural network
simple_rnn_model = simple_model(
    tmp_x.shape,
    preproc_french_sentences.shape[1],
    len(english_tokenizer.word_index),
    len(french_tokenizer.word_index))
simple_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)

# Print prediction(s)
print(logits_to_text(simple_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_5 (LSTM)                (None, 21, 256)           264192    
_________________________________________________________________
time_distributed_3 (TimeDist (None, 21, 344)           88408     
=================================================================
Total params: 352,600
Trainable params: 352,600
Non-trainable params: 0
_________________________________________________________________
[17 23  1  8 67  4 39  7  3  1 55  2 44  0  0]
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_6 (LSTM)                (None, 21, 256)           264192    
_________________________________________________________________
time_distributed_4 (TimeDist (None, 21, 344)           88408     
=================================================================
Total params: 352,600
Trainable params: 352,600
Non-trainable params: 0
_________________________________________________________________
Train on 110288 samples, validate on 27573 samples
Epoch 1/10
110288/110288 [==============================] - 87s - loss: 2.8173 - acc: 0.4496 - val_loss: nan - val_acc: 0.5110
Epoch 2/10
110288/110288 [==============================] - 89s - loss: 1.9995 - acc: 0.5437 - val_loss: nan - val_acc: 0.5793
Epoch 3/10
110288/110288 [==============================] - 86s - loss: 1.7131 - acc: 0.5781 - val_loss: nan - val_acc: 0.5899
Epoch 4/10
110288/110288 [==============================] - 86s - loss: 1.5845 - acc: 0.5925 - val_loss: nan - val_acc: 0.5931
Epoch 5/10
110288/110288 [==============================] - 84s - loss: 1.4995 - acc: 0.6019 - val_loss: nan - val_acc: 0.5908
Epoch 6/10
110288/110288 [==============================] - 84s - loss: 1.4370 - acc: 0.6087 - val_loss: nan - val_acc: 0.5914
Epoch 7/10
110288/110288 [==============================] - 82s - loss: 1.3883 - acc: 0.6154 - val_loss: nan - val_acc: 0.5887
Epoch 8/10
110288/110288 [==============================] - 82s - loss: 1.3500 - acc: 0.6206 - val_loss: nan - val_acc: 0.5851
Epoch 9/10
110288/110288 [==============================] - 82s - loss: 1.3180 - acc: 0.6262 - val_loss: nan - val_acc: 0.5838
Epoch 10/10
110288/110288 [==============================] - 81s - loss: 1.2914 - acc: 0.6299 - val_loss: nan - val_acc: 0.5831
new jersey est parfois agréable en printemps mais il est agréable en en <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>

Model 2: Embedding (IMPLEMENTATION)

RNN You've turned the words into ids, but there's a better representation of a word. This is called word embeddings. An embedding is a vector representation of the word that is close to similar words in n-dimensional space, where the n represents the size of the embedding vectors.

In this model, you'll create a RNN model using embedding.

In [91]:
from keras.layers.embeddings import Embedding


def embed_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
    """
    Build and train a RNN model using word embedding on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param english_vocab_size: Number of unique English words in the dataset
    :param french_vocab_size: Number of unique French words in the dataset
    :return: Keras model built, but not trained
    """
    print(input_shape,input_shape[1:],output_sequence_length)
    model = Sequential()
    model.add(Embedding(english_vocab_size, french_vocab_size, input_shape=input_shape[1:]))
    model.add(LSTM(256, dropout=0.2, recurrent_dropout=0.2, return_sequences=True))
    model.add(Dense(french_vocab_size, activation="softmax"))
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    model.summary()

    return model
tests.test_embed_model(embed_model)


# TODO: Reshape the input
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2]))

# TODO: Train the neural network
curr_embed_model = embed_model(
    tmp_x.shape,
    preproc_french_sentences.shape[1],
    len(english_tokenizer.word_index),
    len(french_tokenizer.word_index))
curr_embed_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)

# TODO: Print prediction(s)
print(logits_to_text(curr_embed_model.predict(tmp_x[:1])[0], french_tokenizer))
(137861, 21) (21,) 21
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_35 (Embedding)     (None, 21, 344)           68456     
_________________________________________________________________
lstm_40 (LSTM)               (None, 21, 256)           615424    
_________________________________________________________________
dense_29 (Dense)             (None, 21, 344)           88408     
=================================================================
Total params: 772,288
Trainable params: 772,288
Non-trainable params: 0
_________________________________________________________________
[17 23  1  8 67  4 39  7  3  1 55  2 44  0  0]
(137861, 21) (21,) 21
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_36 (Embedding)     (None, 21, 344)           68456     
_________________________________________________________________
lstm_41 (LSTM)               (None, 21, 256)           615424    
_________________________________________________________________
dense_30 (Dense)             (None, 21, 344)           88408     
=================================================================
Total params: 772,288
Trainable params: 772,288
Non-trainable params: 0
_________________________________________________________________
Train on 110288 samples, validate on 27573 samples
Epoch 1/10
110288/110288 [==============================] - 123s - loss: 3.0085 - acc: 0.4498 - val_loss: nan - val_acc: 0.5264
Epoch 2/10
110288/110288 [==============================] - 116s - loss: 1.6392 - acc: 0.6079 - val_loss: nan - val_acc: 0.6981
Epoch 3/10
110288/110288 [==============================] - 121s - loss: 1.0170 - acc: 0.7449 - val_loss: nan - val_acc: 0.7964
Epoch 4/10
110288/110288 [==============================] - 126s - loss: 0.6865 - acc: 0.8045 - val_loss: nan - val_acc: 0.8341
Epoch 5/10
110288/110288 [==============================] - 126s - loss: 0.5456 - acc: 0.8332 - val_loss: nan - val_acc: 0.8586
Epoch 6/10
110288/110288 [==============================] - 125s - loss: 0.4655 - acc: 0.8538 - val_loss: nan - val_acc: 0.8750
Epoch 7/10
110288/110288 [==============================] - 124s - loss: 0.4130 - acc: 0.8681 - val_loss: nan - val_acc: 0.8864
Epoch 8/10
110288/110288 [==============================] - 124s - loss: 0.3742 - acc: 0.8793 - val_loss: nan - val_acc: 0.8974
Epoch 9/10
110288/110288 [==============================] - 123s - loss: 0.3440 - acc: 0.8884 - val_loss: nan - val_acc: 0.9037
Epoch 10/10
110288/110288 [==============================] - 123s - loss: 0.3217 - acc: 0.8952 - val_loss: nan - val_acc: 0.9095
new jersey est parfois calme en l' et l' est neigeux neigeux avril avril avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>

Model 3: Bidirectional RNNs (IMPLEMENTATION)

RNN One restriction of a RNN is that it can't see the future input, only the past. This is where bidirectional recurrent neural networks come in. They are able to see the future data.

In [8]:
from keras.layers import Bidirectional
from keras.layers import GRU, Input, Dense, TimeDistributed,LSTM
from keras.models import Model, Sequential
from keras.layers import Activation
from keras.optimizers import Adam
from keras.losses import sparse_categorical_crossentropy

learning_rate=.001

def bd_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
    """
    Build and train a bidirectional RNN model on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param english_vocab_size: Number of unique English words in the dataset
    :param french_vocab_size: Number of unique French words in the dataset
    :return: Keras model built, but not trained
    """
    # TODO: Implement
    model = Sequential()
#     model.add(Embedding(english_vocab_size, french_vocab_size,input_shape=input_shape[1:]))
#     model.add(Input(shape=input_shape[1:]))
    model.add(Bidirectional(LSTM(256, dropout=0.2, recurrent_dropout=0.2, return_sequences=True),input_shape=input_shape[1:]))
    model.add(Dense(french_vocab_size, activation="softmax"))
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    model.summary()
    return model
tests.test_bd_model(bd_model)


# TODO: Train and Print prediction(s)
# Reshaping the input to work with a basic RNN
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))

# Train the neural network
curr_bd_model = bd_model(
    tmp_x.shape,
    preproc_french_sentences.shape[1],
    len(english_tokenizer.word_index),
    len(french_tokenizer.word_index))
curr_bd_model.fit(tmp_x, preproc_french_sentences, batch_size=12, epochs=10, validation_split=0.2)

# Print prediction(s)
print(logits_to_text(curr_bd_model.predict(tmp_x[:1])[0], french_tokenizer))
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
bidirectional_1 (Bidirection (None, 21, 512)           528384    
_________________________________________________________________
dense_1 (Dense)              (None, 21, 344)           176472    
=================================================================
Total params: 704,856
Trainable params: 704,856
Non-trainable params: 0
_________________________________________________________________
[17 23  1  8 67  4 39  7  3  1 55  2 44  0  0]
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
bidirectional_2 (Bidirection (None, 21, 512)           528384    
_________________________________________________________________
dense_2 (Dense)              (None, 21, 344)           176472    
=================================================================
Total params: 704,856
Trainable params: 704,856
Non-trainable params: 0
_________________________________________________________________
Train on 110288 samples, validate on 27573 samples
Epoch 1/10
110288/110288 [==============================] - 1411s - loss: 1.2275 - acc: 0.6449 - val_loss: nan - val_acc: 0.5840 - ETA: 933s - loss: 1.4561 - acc: 0.6119
Epoch 2/10
110288/110288 [==============================] - 1316s - loss: 1.0421 - acc: 0.6734 - val_loss: nan - val_acc: 0.5818 - ETA: 1122s - loss: 1.0746 - acc: 0.6676 - ETA: 732s - loss: 1.0586 - acc: 0.6709
Epoch 3/10
110288/110288 [==============================] - 1211s - loss: 0.9949 - acc: 0.6804 - val_loss: nan - val_acc: 0.5711
Epoch 4/10
110288/110288 [==============================] - 1148s - loss: 0.9643 - acc: 0.6855 - val_loss: nan - val_acc: 0.5666
Epoch 5/10
110288/110288 [==============================] - 1146s - loss: 0.9422 - acc: 0.6896 - val_loss: nan - val_acc: 0.5699
Epoch 6/10
110288/110288 [==============================] - 1146s - loss: 0.9252 - acc: 0.6937 - val_loss: nan - val_acc: 0.5771
Epoch 7/10
110288/110288 [==============================] - 1150s - loss: 0.9091 - acc: 0.6985 - val_loss: nan - val_acc: 0.5625
Epoch 8/10
110288/110288 [==============================] - 1150s - loss: 0.8916 - acc: 0.7036 - val_loss: nan - val_acc: 0.5598
Epoch 9/10
110288/110288 [==============================] - 1149s - loss: 0.8780 - acc: 0.7080 - val_loss: nan - val_acc: 0.5651
Epoch 10/10
110288/110288 [==============================] - 1149s - loss: 0.8655 - acc: 0.7111 - val_loss: nan - val_acc: 0.5690
new jersey est parfois merveilleux en juin mais il est agréable en mars <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>

Model 4: Encoder-Decoder (OPTIONAL)

Time to look at encoder-decoder models. This model is made up of an encoder and decoder. The encoder creates a matrix representation of the sentence. The decoder takes this matrix as input and predicts the translation as output.

Create an encoder-decoder model in the cell below.

In [ ]:
from keras.layers import RepeatVector


def encdec_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
    """
    Build and train an encoder-decoder model on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param english_vocab_size: Number of unique English words in the dataset
    :param french_vocab_size: Number of unique French words in the dataset
    :return: Keras model built, but not trained
    """
    # OPTIONAL: Implement
    return None
tests.test_encdec_model(encdec_model)


# OPTIONAL: Train and Print prediction(s)

Model 5: Custom (IMPLEMENTATION)

Use everything you learned from the previous models to create a model that incorporates embedding and a bidirectional rnn into one model.

In [116]:
def model_final(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
    """
    Build and train a model that incorporates embedding, encoder-decoder, and bidirectional RNN on x and y
    :param input_shape: Tuple of input shape
    :param output_sequence_length: Length of output sequence
    :param english_vocab_size: Number of unique English words in the dataset
    :param french_vocab_size: Number of unique French words in the dataset
    :return: Keras model built, but not trained
    """
    # TODO: Implement
    model = Sequential()
    model.add(Embedding(english_vocab_size, french_vocab_size,input_shape=input_shape[1:]))
#     model.add(Input(shape=input_shape[1:]))
    model.add(Bidirectional(LSTM(256, dropout=0.2, recurrent_dropout=0.2, return_sequences=True)))
    model.add(Dense(french_vocab_size, activation="softmax"))
#     model.add(Dense(french_vocab_size, activation="softmax"))
    model.compile(loss=sparse_categorical_crossentropy,
                  optimizer=Adam(learning_rate),
                  metrics=['accuracy'])
    model.summary()
    return model

# tests.test_model_final(model_final)


# TODO: Reshape the input
tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])
tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2]))

# TODO: Train the neural network
curr_model_final = model_final(
    tmp_x.shape,
    preproc_french_sentences.shape[1],
    len(english_tokenizer.word_index),
    len(french_tokenizer.word_index))
curr_model_final.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)

# TODO: Print prediction(s)
print(logits_to_text(curr_model_final.predict(tmp_x[:1])[0], french_tokenizer))
print('Final Model Loaded')
[17 23  1  8 67  4 39  7  3  1 55  2 44  0  0]
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_41 (Embedding)     (None, 21, 344)           68456     
_________________________________________________________________
bidirectional_14 (Bidirectio (None, 21, 512)           1230848   
_________________________________________________________________
dense_45 (Dense)             (None, 21, 344)           176472    
=================================================================
Total params: 1,475,776
Trainable params: 1,475,776
Non-trainable params: 0
_________________________________________________________________
Train on 110288 samples, validate on 27573 samples
Epoch 1/10
110288/110288 [==============================] - 285s - loss: 2.6605 - acc: 0.4814 - val_loss: nan - val_acc: 0.5606
Epoch 2/10
110288/110288 [==============================] - 293s - loss: 1.4148 - acc: 0.6511 - val_loss: nan - val_acc: 0.7418
Epoch 3/10
110288/110288 [==============================] - 265s - loss: 0.8049 - acc: 0.7848 - val_loss: nan - val_acc: 0.8306
Epoch 4/10
110288/110288 [==============================] - 258s - loss: 0.5366 - acc: 0.8406 - val_loss: nan - val_acc: 0.8712
Epoch 5/10
110288/110288 [==============================] - 483s - loss: 0.4091 - acc: 0.8733 - val_loss: nan - val_acc: 0.8999
Epoch 6/10
110288/110288 [==============================] - 374s - loss: 0.3333 - acc: 0.8956 - val_loss: nan - val_acc: 0.9173
Epoch 7/10
110288/110288 [==============================] - 337s - loss: 0.2821 - acc: 0.9108 - val_loss: nan - val_acc: 0.9312
Epoch 8/10
110288/110288 [==============================] - 330s - loss: 0.2450 - acc: 0.9227 - val_loss: nan - val_acc: 0.9403
Epoch 9/10
110288/110288 [==============================] - 311s - loss: 0.2157 - acc: 0.9322 - val_loss: nan - val_acc: 0.9474
Epoch 10/10
110288/110288 [==============================] - 318s - loss: 0.1955 - acc: 0.9384 - val_loss: nan - val_acc: 0.9520
new jersey est parfois calme pendant l' et il est neigeux en avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>
Final Model Loaded

Prediction (IMPLEMENTATION)

In [119]:
import numpy as np
from keras.preprocessing.sequence import pad_sequences


def final_predictions(x, y, x_tk, y_tk):
    """
    Gets predictions using the final model
    :param x: Preprocessed English data
    :param y: Preprocessed French data
    :param x_tk: English tokenizer
    :param y_tk: French tokenizer
    """
    
    
    # TODO: Reshape the input
    x = pad(x, y.shape[1])
    x = tmp_x.reshape((-1, y.shape[-2]))


    
    # TODO: Train neural network using model_final
    model = model_final(
        x.shape,
        y.shape[1],
        len(x_tk.word_index),
        len(y_tk.word_index))
    
    model.fit(x, y, batch_size=1024, epochs=25, validation_split=0.2)

    
    ## DON'T EDIT ANYTHING BELOW THIS LINE
    y_id_to_word = {value: key for key, value in y_tk.word_index.items()}
    y_id_to_word[0] = '<PAD>'

    sentence = 'he saw a old yellow truck'
    sentence = [x_tk.word_index[word] for word in sentence.split()]
    sentence = pad_sequences([sentence], maxlen=x.shape[-1], padding='post')
    sentences = np.array([sentence[0], x[0]])
    predictions = model.predict(sentences, len(sentences))

    print('Sample 1:')
    print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[0]]))
    print('Il a vu un vieux camion jaune')
    print('Sample 2:')
    print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[1]]))
    print(' '.join([y_id_to_word[np.argmax(x)] for x in y[0]]))


final_predictions(preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer)
[17 23  1  8 67  4 39  7  3  1 55  2 44  0  0]
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_44 (Embedding)     (None, 21, 344)           68456     
_________________________________________________________________
bidirectional_17 (Bidirectio (None, 21, 512)           1230848   
_________________________________________________________________
dense_48 (Dense)             (None, 21, 344)           176472    
=================================================================
Total params: 1,475,776
Trainable params: 1,475,776
Non-trainable params: 0
_________________________________________________________________
Train on 110288 samples, validate on 27573 samples
Epoch 1/25
110288/110288 [==============================] - 232s - loss: 2.6649 - acc: 0.4813 - val_loss: nan - val_acc: 0.5594
Epoch 2/25
110288/110288 [==============================] - 233s - loss: 1.4128 - acc: 0.6503 - val_loss: nan - val_acc: 0.7449
Epoch 3/25
110288/110288 [==============================] - 234s - loss: 0.7960 - acc: 0.7874 - val_loss: nan - val_acc: 0.8332
Epoch 4/25
110288/110288 [==============================] - 235s - loss: 0.5246 - acc: 0.8430 - val_loss: nan - val_acc: 0.8738
Epoch 5/25
110288/110288 [==============================] - 236s - loss: 0.3995 - acc: 0.8753 - val_loss: nan - val_acc: 0.9023
Epoch 6/25
110288/110288 [==============================] - 235s - loss: 0.3263 - acc: 0.8976 - val_loss: nan - val_acc: 0.9192
Epoch 7/25
110288/110288 [==============================] - 237s - loss: 0.2774 - acc: 0.9125 - val_loss: nan - val_acc: 0.9289
Epoch 8/25
110288/110288 [==============================] - 234s - loss: 0.2401 - acc: 0.9244 - val_loss: nan - val_acc: 0.9410
Epoch 9/25
110288/110288 [==============================] - 231s - loss: 0.2123 - acc: 0.9332 - val_loss: nan - val_acc: 0.9483
Epoch 10/25
110288/110288 [==============================] - 231s - loss: 0.1903 - acc: 0.9405 - val_loss: nan - val_acc: 0.9536
Epoch 11/25
110288/110288 [==============================] - 231s - loss: 0.1732 - acc: 0.9459 - val_loss: nan - val_acc: 0.9587
Epoch 12/25
110288/110288 [==============================] - 231s - loss: 0.1593 - acc: 0.9502 - val_loss: nan - val_acc: 0.9601
Epoch 13/25
110288/110288 [==============================] - 231s - loss: 0.1467 - acc: 0.9542 - val_loss: nan - val_acc: 0.9646
Epoch 14/25
110288/110288 [==============================] - 231s - loss: 0.1372 - acc: 0.9571 - val_loss: nan - val_acc: 0.9658
Epoch 15/25
110288/110288 [==============================] - 231s - loss: 0.1279 - acc: 0.9601 - val_loss: nan - val_acc: 0.9676
Epoch 16/25
110288/110288 [==============================] - 231s - loss: 0.1210 - acc: 0.9620 - val_loss: nan - val_acc: 0.9698
Epoch 17/25
110288/110288 [==============================] - 231s - loss: 0.1147 - acc: 0.9640 - val_loss: nan - val_acc: 0.9711
Epoch 18/25
110288/110288 [==============================] - 231s - loss: 0.1099 - acc: 0.9655 - val_loss: nan - val_acc: 0.9724
Epoch 19/25
110288/110288 [==============================] - 231s - loss: 0.1044 - acc: 0.9671 - val_loss: nan - val_acc: 0.9736
Epoch 20/25
110288/110288 [==============================] - 231s - loss: 0.0998 - acc: 0.9685 - val_loss: nan - val_acc: 0.9745
Epoch 21/25
110288/110288 [==============================] - 231s - loss: 0.0959 - acc: 0.9696 - val_loss: nan - val_acc: 0.9747
Epoch 22/25
110288/110288 [==============================] - 231s - loss: 0.0924 - acc: 0.9709 - val_loss: nan - val_acc: 0.9758
Epoch 23/25
110288/110288 [==============================] - 231s - loss: 0.0885 - acc: 0.9719 - val_loss: nan - val_acc: 0.9768
Epoch 24/25
110288/110288 [==============================] - 231s - loss: 0.0856 - acc: 0.9729 - val_loss: nan - val_acc: 0.9767
Epoch 25/25
110288/110288 [==============================] - 231s - loss: 0.0829 - acc: 0.9736 - val_loss: nan - val_acc: 0.9779
Sample 1:
il a vu un vieux camion jaune <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>
Il a vu un vieux camion jaune
Sample 2:
new jersey est parfois calme pendant l' automne et il est neigeux en avril <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>
<PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>
In [ ]:
 

Submission

When you are ready to submit your project, do the following steps:

  1. Ensure you pass all points on the rubric.
  2. Submit the following in a zip file.
    • helper.py
    • machine_translation.ipynb
    • machine_translation.html
      • You can export the notebook by navigating to File -> Download as -> HTML (.html).